Testing

This test has been run with Omega on chrysalis (intel, openmpi, cpu) but doesn't produce meaningful results in the absence of wind forcing.

@mark-petersen Can you suggest the changes you had to make to get Omega to run properly? Or if it's easier you can check out my branch, make changes, and open a new PR and I'll close this one. Thanks!

Thanks! I am happy to make polaris alterations after I return, but realistically it will be after May 19. Here are the changes I needed to run the barotropic gyre case:

omega.yml file

pwd
/pscratch/sd/m/mpeterse/runs/250502-omega-barotropic_gyre/ocean/planar/barotropic_gyre/default/long_forward
(dev_compass_1.7.0-alpha.2) pm:long_forward$ diff orig/omega.yml omega.yml
4c4
<     TimeStepper: RK4
---
>     TimeStepper: RungeKutta4
28a29,31
>     WindForcingTendencyEnable: true
>     BottomDragTendencyEnable: true
>     BottomDragCoeff: 0.001
40,41c43,70
<     Base: [Temperature, Salinity]
<     Debug: [Debug1, Debug2, Debug3]
---
>   IOStreams:
>     # InitialState should only be used when starting from scratch.
>     # For restart runs, the frequency units should be changed from
>     # "OnStartup" to "never" so that the initial state file is not read.
>     InitialState:
>       UsePointerFile: false
>       Filename: init_with_wind.nc
>       Mode: read
>       Precision: double
>       Freq: 1
>       FreqUnits: OnStartup
>       UseStartEnd: false
>       Contents:
>         - State
>         - WindStressZonal
>         - WindStressMeridional
>     History:
>       UsePointerFile: false
>       Filename: ocn.hist.$SimTime
>       Mode: write
>       IfExists: replace
>       Precision: double
>       Freq: 1
>       FreqUnits: months
>       UseStartEnd: false
>       Contents:
>         - State
>         - SshCell

Add wind fields to init.nc

ncks -A -v windStressZonal,windStressMeridional forcing.nc init_with_wind.nc

This could also be accomplished by adding a new forcing stream to omega.yml:

ADD FORCING STREAM:
>     Forcing:
>       UsePointerFile: false
>       Filename: forcing.nc
>       Mode: read
>       Precision: double
>       Freq: 1
>       FreqUnits: OnStartup
>       UseStartEnd: false
>       Contents:
>         - WindStressZonal
>         - WindStressMeridional

but it requires an addition to the code. From Brian O'Neill:

Also, the code needs a statement to read in the Forcing stream. I added Err = IOStream::read("Forcing", ModelClock, ReqMeta); to src/ocn/OceanInit.cpp after the InitialState stream is read in at line 182, and that seems to be enough to get it working. When I run and look at the output, looks like it's actually doing stuff now, and it's not just NaNs and 0s.

output alteration for polaris analysis step

/pscratch/sd/m/mpeterse/runs/250502-omega-barotropic_gyre/ocean/planar/barotropic_gyre/default/analysis
cp ../long_forward/ocn.hist.0004-01-01_00\:00\:48.nc output.nc
ncrename -d time,Time output.nc
ncrename -d NVertLevels,nVertLevels output.nc
ncrename -v LayerThickness,layerThickness output.nc
ncrename -d NCells,nCells output.nc
ncrename -v NormalVelocity,normalVelocity output.nc
ncrename -d NEdges,nEdges output.nc
# note: on login node, not interactive. That hangs.
polaris serial

That was just changing camel case to Pascal case. That is a general need for all analysis going forward.

@mark-petersen I added this commit 2c82f97 to reflect #286 (comment). Since we are still waiting on Omega forcing stream and further changes will be made, this commit is untested.

I moved the forcing to the initial state file and successfully ran the short_forward step with omega. This PR will remain in draft mode until E3SM-Project/Omega#219 is merged.

Testing

Free-slip results generated with chrys, intel-openmpi are shown here: E3SM-Ocean-Discussion/E3SM#49 (comment)

@xylar would you like to review this pr? Happy to ask others if you have too much on your plate right now.

Outstanding issue: steps to run appears in section munk/no-slip rather than barotropic_gyre_munk_no-slip and similarly for free-slip.

@cbegeman, I can review but I have somewhat limited ability to test right now because I don't have Chrysalis access. I'll put it on my to-do list for tomorrow.

Hi @mwarusz or @mark-petersen, Do either of you have time to look into the fail I'm getting with Omega for this wind-forced test case? I think it's likely related to wind forcing but I'm not sure if the issue is arising because I am applying the forcing incorrectly or due to something internal to that Omega tendency. The test can be found on frontier at /lustre/orion/cli115/scratch/cbegeman/polaris-tasks/fixup-and-enhance-btrgyre-tests-v0/omega/1fa63b9

Hi @mwarusz or @mark-petersen, Do either of you have time to look into the fail I'm getting with Omega for this wind-forced test case? I think it's likely related to wind forcing but I'm not sure if the issue is arising because I am applying the forcing incorrectly or due to something internal to that Omega tendency. The test can be found on frontier at /lustre/orion/cli115/scratch/cbegeman/polaris-tasks/fixup-and-enhance-btrgyre-tests-v0/omega/1fa63b9

@cbegeman
What exactly is the fail ? I am getting "Permission denied" for the directory you posted.

@mwarusz Thanks for looking into it! I changed the permissions. Let me know if you still have trouble accessing it.

The solution is NaNs sometime after 3 time steps (the duration of the short run) but before 1 month (the output interval of the long run). I tried decreasing the time step with the same results.

@cbegeman

The solution is NaNs sometime after 3 time steps (the duration of the short run) but before 1 month (the output interval of the long run). I tried decreasing the time step with the same results.

What was the smallest time step you tried ? I was able to get stable 2-year runs with 50 s time step (1 min was unstable). I ran this test case using MPAS-O with RK4 and I also had to use 50 s time step to get stable results.

I got interested in seeing how close can I make the Omega and MPAS-O solutions for this test. To do that I changed the MPAS-O config to add nonlinear advection terms (we don't have linearlized shallow-water tendencies in Omega) and made sure that the relevant constants agree between the models. With that I was able to get the following figure showing the normalized difference between the models:

The agreement is pretty good: 1e-14 for layer thickness and 1e-10 for velocities.

@mwarusz Great! Thanks for testing. I ran the omega RK4 test with dt = 00:01:02. That actually is consistent with the barotropic time step I had used for the MPAS-O test. I'll make changes to the test case config so that the RK4 time step is set to the barotropic time step.

Update regarding Omega support

I am currently investigating why Omega seems to have an offset in the y-direction.

The time integrator must also be manually changed by the user to RK4 in the cfg file after test case setup. I retain the default of AB2 so that MPAS-O tests run more quickly.

@brian-oneill Thanks for offering to help with this! The variable that I would need in the output is called RelVortVertex

I rebased onto #384 and this test case still passes. I added it to the omega_pr suite even though the solution isn't as close to the analytic solution as one would like. It still seemed better to increase the number of PR tests that might break when Omega needs fixing.

@xylar Ready for your review when you have time. I'll wait to merge until after #384.